Education Day 2025

Feedback machine

Caspar David Peter
peter@rsm.nl

Rotterdam School of Management

Feedback machine

Disclaimer

This is a side-hustle of mine!

I do this for fun and to learn new things, at night and on weekends!

It reflects my personal views and not those of RSM!

If I bring this thing into production you can find it on my GitHub page

There are more efficient ways to build certain parts of this machine, but I want to keep it open source and easy to understand!

Feedback machine

Big picture

Giving feedback is, …

  • time consuming
  • cumbersome
  • not always constructive?
    • Garbage in, garbage out
  • inconsistent

Feedback machine

Work smarter, not harder

Benefits

  • automated
  • constructive
  • efficient
  • scalable
  • customizable
  • professor-friendly

Feedback machine

Current state

Drawbacks

  • Needs human input
    • Still a good starting point!
  • Not scalable, yet
  • Not easily deployable

Feedback machine

How does it work, so far…

Feedback machine

Demo (kind of)

Research idea

User Input

How does the introduction of IFRS affect earnings management?

Vector search results
Output

Feedback machine

How it’s going…

Inputs

  • Uses only abstracts for now
  • Not all abstracts are available on SSRN
  • Correct citation format has to be given
  • More prompts & more engineering needed
Next steps
  • Utilize full PDFs
  • Requires revamp of inner workings of the machine, e.g prompt(s)
  • Agentic1 workflow?

Outputs

  • Citation & references not always consistent
  • Still minor bugs in the formatting
  • Output not deterministic
  • Feedback is shallow at times
Next steps
  • Introduce Human in the loop?
  • Develop prompts for complete drafts
  • Consider using LangSmith2 for quality control

Feedback machine

Where to go from here?

Discussion